Imitation and Reinforcement Learning Practical Learning Algorithms for Motor Primitives in Robotics

TO date, most robots are still programmed by a smart operator who uses human understanding of the desired task to create a program for accomplishing the required behavior. While such specialized programming is highly efficient, it is also expensive and limited to the situations the human operator had considered. For example, human programming has become the main bottleneck for manufacturing of low-cost products in low numbers. This problem could be alleviated by robots that can learn new skills and improve their existing abilities autonomously. However, off-the-shelf machine learning techniques do not scale to high-dimensional, anthropomorphic robots. Instead, robot learning requires methods that employ both representations and algorithms appropriate for this domain. When humans learn new motor skills, e.g., paddling a ball with a tabletennis racket, throwing darts, or hitting a tennis ball, it is highly likely that they rely on a small set of motor primitives (MPs) and use imitation as well as reinforcement learning (RL) [1]. Inspired by this example, we will discuss the technical counterparts in this article and show how both single-stroke and rhythmic tasks can be learned efficiently by mimicking the human presenter with subsequent reward-driven self-improvement. Recently, the idea of using dynamical systems as MPs was put forward by Ijspeert et al. [2] as a general approach of representing control policies for basic movements. The resulting movement generation has a variety of favorable properties, i.e., basic stability properties, the ability to encode either single-stroke or rhythmic behaviors, as well as rescalability with respect to time, goal, and amplitude. As this representation is linear in its parameters, learning can be sufficiently fast for real-time applications in robotics. A series of such frameworks has been introduced to date [2], [3]. Previous applications include a variety of different basic motor skills such as tennis swings [2], T-ball batting [4], drumming [5], planar biped walking [6], constrained reaching tasks [7] and even in tasks with potential industrial appli-